Fix upstream map index resolution after placeholder expansion#59691
Fix upstream map index resolution after placeholder expansion#59691SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Conversation
SameerMesiah97
force-pushed
the
59289-DTM-Placeholder-Resolution
branch
from
79528c5 to
529673d
Compare
|
Also @uranusjr can help with review - there ias related WIP he eworks on now I think |
SameerMesiah97
force-pushed
the
59289-DTM-Placeholder-Resolution
branch
2 times, most recently
from
3b16811 to
cee1968
Compare
SameerMesiah97
force-pushed
the
59289-DTM-Placeholder-Resolution
branch
from
cee1968 to
630de66
Compare
|
I have checked the main branch using a variety of DAG shapes following the same theme as the one in the bug report and the issue is still present. So it appears any recent work in this specific area (that is separate from this PR) which has been merged into main, has not resolved the bug. As of this comment, I cannot see any open PRs that could resolve the issue (anyone reading this comment is free to correct me). This PR has been rebased and is ready for review. |
eladkal
added
area:core
area:dynamic-task-mapping
SameerMesiah97
force-pushed
the
59289-DTM-Placeholder-Resolution
branch
from
630de66 to
4bff1ae
Compare
|
I have checked main again using the same test cases and the issue is still present so the bug being addressed by this PR has not been resolved. I am unable to locate any open PRs that I would expect to fix this issue either. The bug silently renders certain DAG shapes unusable with Dynamic Task Mapping so I believe this should not be deprioritized. This PR has been open for approximately 2 months. It has now been rebased again and is ready for review. I have run the aforementioned test cases with my changes and I can confirm that they give the expected result. |
4 participants
Description
Fix upstream map index resolution for dynamically mapped task groups when placeholder task instances are replaced during expansion.
After expansion, the upstream placeholder (
map_index = -1) is replaced by the first expanded task instance (map_index = 0). Downstream task instances that are still unexpanded may continue to reference the placeholder, causing incorrect upstream dependency resolution.This change introduces a small helper (
resolve_placeholder_map_index) that is invoked during upstream map index resolution to correctly handle this transition, while preserving existing behavior for already-expanded downstream tasks.Rationale
Placeholder task instances are a temporary pre-expansion representation. Once expansion occurs, they are no longer valid references, but downstream resolution logic may still encounter them. Handling this transition explicitly prevents downstream tasks from treating completed upstream work as unresolved.
This could cause implicit dependencies to be skipped during DAG evaluation under certain trigger rules (such as
NONE_FAILED_MIN_ONE_SUCCESS), particularly when multiple parallel task streams within theMappedTaskGroupconverge on a single downstream task outside the group. As a result, DAG runs could be incorrectly marked as failed or skipped despite valid upstream execution.Please refer to issue #59289 for more context. This PR was opened in response to that. The author of the issue reported the bug in Airflow 3.0.6 but I can confirm that the same issue is present in Airflow 3.1.5 (as well as the main branch at the time of opening this PR).
Tests
Added a test covering the post-expansion state where:
The test asserts that
get_relevant_upstream_map_indexesreturns the correct upstream map index in both cases.Closes: #59289